Grafana automation.md
----------------------


## ✅ Yes — You Can Fully Automate the Entire Stack

And with your existing setup (Docker on Hetzner, Activepieces, Telegram bots), you're already 70% of the way there. The architecture is:

```
┌─────────────────┐     ┌──────────────────┐     ┌─────────────────────┐
│   AI Agent      │────►│  Grafana API     │────►│  Prometheus + Loki  │
│  (Controller)   │     │  + Provisioning  │     │  + Node Exporter   │
└─────────────────┘     └──────────────────┘     └─────────────────────┘
         │                                               │
         └───────────────────────────────────────────────┘
                              ▲
                    Feedback loop: metrics → AI decision
```

---

## 🏗️ Layer 1: One-Shot Infrastructure Deployment

Deploy everything via Docker Compose with **pre-wired connections**. Grafana will auto-connect to Prometheus and Loki on first boot.

```yaml
# /root/observability/docker-compose.yml
version: "3.8"

services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=15d'
    networks: [observability]

  loki:
    image: grafana/loki:latest
    volumes:
      - ./loki-config.yml:/etc/loki/local-config.yaml
      - loki_data:/loki
    command: -config.file=/etc/loki/local-config.yaml
    networks: [observability]

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASS}
      - GF_INSTALL_PLUGINS=grafana-clock-panel,grafana-simple-json-datasource
    volumes:
      - grafana_data:/var/lib/grafana
      # 🔥 This is the magic: provisioning files auto-configure on boot
      - ./provisioning/datasources:/etc/grafana/provisioning/datasources
      - ./provisioning/dashboards:/etc/grafana/provisioning/dashboards
      - ./provisioning/alerting:/etc/grafana/provisioning/alerting
    networks: [observability]
    depends_on: [prometheus, loki]

  node-exporter:
    image: prom/node-exporter:latest
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    networks: [observability]

  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - ./promtail-config.yml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml
    networks: [observability]

volumes:
  prometheus_data:
  loki_data:
  grafana_data:

networks:
  observability:
    driver: bridge
```

---

## ⚙️ Layer 2: Configuration as Code (Provisioning Files)

Grafana's **provisioning directory** is more reliable than API for initial setup because it survives container restarts.

### 1. Auto-Connect Data Sources
```yaml
# provisioning/datasources/datasources.yml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false

  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    editable: false

  - name: Alertmanager
    type: alertmanager
    url: http://alertmanager:9093
    editable: false
```

### 2. Auto-Import Dashboards
```yaml
# provisioning/dashboards/dashboards.yml
apiVersion: 1

providers:
  - name: 'default'
    folder: 'Server Metrics'
    type: file
    options:
      path: /var/lib/grafana/dashboards
```

Download these JSONs into `./dashboards/` before first boot:
- **Node Exporter Full:** ID `1860` from [grafana.com/dashboards](https://grafana.com/dashboards)
- **Docker Monitoring:** ID `193`
- **Loki Logs:** ID `13639`

### 3. Auto-Configure Alerts (Grafana 11+)
```yaml
# provisioning/alerting/contact-points.yml
apiVersion: 1

contactPoints:
  - orgId: 1
    name: telegram-alerts
    receivers:
      - uid: telegram-1
        type: telegram
        settings:
          bottoken: "${TELEGRAM_BOT_TOKEN}"
          chatid: "${TELEGRAM_CHAT_ID}"
```

```yaml
# provisioning/alerting/rules.yml
apiVersion: 1

groups:
  - orgId: 1
    name: server-alerts
    folder: Infrastructure
    interval: 60s
    rules:
      - uid: ram-alert
        title: RAM Usage > 85%
        condition: B
        data:
          - refId: A
            relativeTimeRange:
              from: 300
              to: 0
            datasourceUid: prometheus
            model:
              expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100
          - refId: B
            datasourceUid: __expr__
            model:
              type: threshold
              expression: A
              conditions:
                - evaluator:
                    type: gt
                    params: [85]
        noDataState: NoData
        execErrState: Error
        for: 5m
        annotations:
          summary: "Server RAM is at {{ $values.A }}%"
        labels:
          severity: critical
```

> **Result:** Run `docker compose up -d` once. Grafana boots with Prometheus connected, Loki connected, 3 dashboards imported, and Telegram alerts armed.

---

## 🤖 Layer 3: The AI Agent Controller

Now the advanced part: an AI agent that **modifies** the running Grafana instance via API — adding new dashboards for new services, tuning alerts, investigating anomalies.

### What the Agent Can Do via Grafana API

| Capability | API Endpoint | AI Use Case |
|------------|-------------|-------------|
| Add data source | `POST /api/datasources` | "Monitor my new PocketBase instance" |
| Create dashboard | `POST /api/dashboards/db` | "Build a dashboard for Listmonk email metrics" |
| Update alert | `PUT /api/v1/provisioning/alert-rules` | "Lower RAM threshold to 80%" |
| Add annotation | `POST /api/annotations` | "Mark deployment time on all graphs" |
| Query data | `POST /api/ds/query` | "Is CPU spiking right now?" |
| Export snapshot | `POST /api/snapshots` | "Send me a screenshot of the crash" |

### Python Agent Framework

This script acts as the **AI agent's hands**. It exposes functions that an LLM can call.

```python
# /root/observability/agent/grafana_agent.py
import requests
import json
import os

class GrafanaAgent:
    def __init__(self):
        self.base = os.getenv("GRAFANA_URL", "http://localhost:3000")
        self.key = os.getenv("GRAFANA_API_KEY")  # Service account token
        self.headers = {
            "Authorization": f"Bearer {self.key}",
            "Content-Type": "application/json"
        }

    # ─── TOOL 1: Create Dashboard ───
    def create_dashboard(self, title, panels, folder_uid="general"):
        dashboard = {
            "dashboard": {
                "title": title,
                "panels": panels,
                "timezone": "Africa/Cairo",
                "schemaVersion": 36
            },
            "folderUid": folder_uid,
            "overwrite": True
        }
        r = requests.post(
            f"{self.base}/api/dashboards/db",
            headers=self.headers,
            json=dashboard
        )
        return r.json()

    # ─── TOOL 2: Add Prometheus Query Panel ───
    def add_metric_panel(self, dashboard_uid, title, expr, unit="percent"):
        panel = {
            "title": title,
            "type": "timeseries",
            "targets": [{
                "expr": expr,
                "legendFormat": "{{instance}}",
                "datasource": {"type": "prometheus", "uid": "prometheus"}
            }],
            "fieldConfig": {
                "defaults": {
                    "unit": unit,
                    "custom": {"lineWidth": 2}
                }
            },
            "gridPos": {"h": 8, "w": 12, "x": 0, "y": 0}
        }
        # Fetch → modify → push back
        db = requests.get(
            f"{self.base}/api/dashboards/uid/{dashboard_uid}",
            headers=self.headers
        ).json()
        db["dashboard"]["panels"].append(panel)
        
        return self.create_dashboard(
            db["dashboard"]["title"],
            db["dashboard"]["panels"]
        )

    # ─── TOOL 3: Create Alert Rule ───
    def create_alert(self, name, query, threshold, for_duration="5m"):
        payload = {
            "title": name,
            "ruleGroup": "ai-generated",
            "folderUID": "alerts",
            "orgID": 1,
            "data": [{
                "refId": "A",
                "queryType": "",
                "relativeTimeRange": {"from": 300, "to": 0},
                "datasourceUid": "prometheus",
                "model": {"expr": query}
            }],
            "condition": "A",
            "noDataState": "NoData",
            "execErrState": "Error",
            "for": for_duration
        }
        return requests.post(
            f"{self.base}/api/v1/provisioning/alert-rules",
            headers=self.headers,
            json=payload
        ).json()

    # ─── TOOL 4: Get Current Metrics ───
    def query_current(self, expr):
        r = requests.post(
            f"{self.base}/api/ds/query",
            headers=self.headers,
            json={
                "queries": [{
                    "refId": "A",
                    "expr": expr,
                    "datasource": {"uid": "prometheus", "type": "prometheus"}
                }],
                "from": "now-5m",
                "to": "now"
            }
        )
        return r.json()

# ─── Example: AI Agent Decision Loop ───
if __name__ == "__main__":
    agent = GrafanaAgent()
    
    # AI detects Listmonk container is new → auto-monitor it
    result = agent.create_dashboard(
        title="Listmonk Email Health",
        panels=[{
            "title": "Emails Sent / Hour",
            "type": "timeseries",
            "targets": [{
                "expr": "rate(listmonk_messages_total[1h])",
                "datasource": {"uid": "prometheus"}
            }]
        }]
    )
    print(f"Dashboard created: {result.get('url')}")
```

---

## 🔗 Wiring the AI Agent into Your Existing Stack

You already have **Activepieces** and **Telegram**. Use them as the agent's nervous system.

### Architecture: Activepieces as the Agent Orchestrator

```
[Telegram Message] 
    ↓
[Activepieces Webhook] 
    ↓
[AI Logic: OpenAI/Claude API] 
    ↓
[Activepieces HTTP Request] 
    ↓
[Grafana Agent API (the Python script above)]
    ↓
[Grafana Server]
```

### Activepieces Flow Setup

1. **Trigger:** Webhook or Telegram Bot message
   - You text: *"Monitor my new app on port 8080"*
   
2. **AI Piece:** OpenAI ChatGPT piece with system prompt:
   ```
   You are a DevOps AI. Convert user requests into Grafana API calls.
   Available tools:
   - create_datasource(url, name)
   - create_dashboard(title, panels)
   - create_alert(metric, threshold)
   
   Respond ONLY with JSON: {"tool": "...", "params": {...}}
   ```

3. **HTTP Piece:** POST to `http://localhost:5000/agent` (your Python agent running as a microservice)

4. **Telegram Reply Piece:** Send back the dashboard URL

---

## 🔐 Security & API Authentication

### 1. Create a Service Account (Not Admin User)

In Grafana UI: **Administration > Users and access > Service accounts**

```bash
# Or via API
curl -X POST http://localhost:3000/api/serviceaccounts \
  -H "Content-Type: application/json" \
  -u admin:yourpassword \
  -d '{"name":"ai-agent","role":"Editor"}'
```

> **Why Editor, not Admin?** The AI agent can create dashboards and alerts but cannot delete users or change passwords. If the agent is compromised, damage is contained.

### 2. Store Token in Activepieces Connections

Don't hardcode tokens in flows. Use Activepieces' **Connections** feature to store the Grafana API key securely.

### 3. Network Isolation

```yaml
# In your Docker Compose, don't expose the agent port publicly
services:
  grafana-agent:
    image: your-agent-image
    environment:
      - GRAFANA_API_KEY=${GRAFANA_API_KEY}
    networks: [observability]
    # NO ports exposed to host! Only internal Docker access
```

Activepieces (if running on the same server) accesses it via `http://grafana-agent:5000` internally.

---

## 🧠 Expert Tricks for the AI Agent

| Trick | Implementation |
|-------|---------------|
| **Dashboard UID Strategy** | Force UIDs like `listmonk-health`, `temporal-jobs`. The AI can check if `uid exists` → update, else → create. Prevents duplicates. |
| **Template Variables** | Create one master dashboard with `$job` variable. The AI just changes the variable instead of cloning dashboards. |
| **Annotation API** | When Activepieces triggers a deployment flow, hit `POST /api/annotations` to draw a vertical red line on all graphs at deploy time. |
| **LogQL Generation** | For Loki, the AI can translate *"Show me Listmonk errors from yesterday"* into LogQL: `{job="listmonk"} \|= "ERROR" \| json \| timestamp > "2026-04-28"` |
| **Anomaly Detection Loop** | Cron job every 5 min: Agent queries `stddev_over_time(cpu[1h])`, detects >2σ spikes, auto-creates a temporary dashboard for investigation. |
| **Snapshot Sharing** | Agent hits `POST /api/snapshots` to create a public (but obfuscated) URL of a dashboard state, sends to Telegram for mobile viewing. |

---

## 📦 Recommended: Deploy the Agent as a Docker Service

```yaml
# Add to your observability docker-compose.yml
  grafana-agent:
    build: ./agent
    environment:
      - GRAFANA_URL=http://grafana:3000
      - GRAFANA_API_KEY=${GRAFANA_API_KEY}
      - TELEGRAM_BOT_TOKEN=${TELEGRAM_BOT_TOKEN}
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock  # For auto-discovery
    networks: [observability]
    restart: unless-stopped
```

**Auto-discovery superpower:** The agent can read `docker ps` output, see a new container named `trae-nutrition-backend`, and automatically:
1. Add a Prometheus job for it
2. Create a dashboard with container CPU/RAM
3. Set a "Container down" alert
4. Send Telegram confirmation

---

## ✅ Summary: Your Automation Roadmap

1. **Today:** Replace your current Grafana with the Docker Compose above → boot once, everything auto-connects
2. **Day 2:** Set up provisioning files for datasources + dashboards + alerts
3. **Day 3:** Create a Grafana Service Account token, store in Activepieces
4. **Day 4:** Build the Python agent microservice with the 4 tools above
5. **Day 5:** Connect Activepieces → Telegram → Agent → Grafana
6. **Ongoing:** Text your bot *"Monitor my new app"* and watch the dashboard appear in 10 seconds

**The result:** A self-healing, self-documenting observability stack where the AI is your 24/7 SRE.